Curation

Curation¶

Note: make a copy of this notebook and run the copy to avoid git conflicts in the future

This is the second in a multi-part tutorial on the NWB-Datajoint pipeline used in Loren Frank's lab, UCSF. It demonstrates how to curate the results of spike sorting.

Finish tutorial 0 and tutorial 1 before proceeding.

Let's start by importing the spyglass package, along with a few others.

In [1]:

import os
import numpy as np

import spyglass as nd

import warnings

warnings.simplefilter("ignore", category=DeprecationWarning)
warnings.simplefilter("ignore", category=ResourceWarning)

import os import numpy as np import spyglass as nd import warnings warnings.simplefilter("ignore", category=DeprecationWarning) warnings.simplefilter("ignore", category=ResourceWarning)

[2022-08-01 17:21:49,989][INFO]: Connecting zoldello@lmf-db.cin.ucsf.edu:3306
[2022-08-01 17:21:50,035][INFO]: Connected zoldello@lmf-db.cin.ucsf.edu:3306

/home/zoldello/anaconda3/envs/spyglass/lib/python3.9/site-packages/position_tools/core.py:3: DeprecationWarning: Please use `gaussian_filter1d` from the `scipy.ndimage` namespace, the `scipy.ndimage.filters` namespace is deprecated.
  from scipy.ndimage.filters import gaussian_filter1d

In [14]:

# We also import a bunch of tables so that we can call them easily
from spyglass.common import (
    RawPosition,
    HeadDir,
    Speed,
    LinPos,
    StateScriptFile,
    VideoFile,
    DataAcquisitionDevice,
    CameraDevice,
    Probe,
    DIOEvents,
    ElectrodeGroup,
    Electrode,
    Raw,
    SampleCount,
    LFPSelection,
    LFP,
    LFPBandSelection,
    LFPBand,
    FirFilterParameters,
    IntervalList,
    Lab,
    LabMember,
    LabTeam,
    Institution,
    BrainRegion,
    SensorData,
    Session,
    ExperimenterList,
    Subject,
    Task,
    TaskEpoch,
    Nwbfile,
    AnalysisNwbfile,
    NwbfileKachery,
    AnalysisNwbfileKachery,
)

from spyglass.spikesorting import (
    ArtifactDetection,
    ArtifactDetectionParameters,
    ArtifactDetectionSelection,
    ArtifactRemovedIntervalList,
    SortGroup,
    SortInterval,
    SpikeSortingPreprocessingParameters,
    SpikeSortingRecording,
    SpikeSortingRecordingSelection,
    SpikeSorterParameters,
    SpikeSorting,
    SpikeSortingSelection,
    SortInterval,
    SpikeSorting,
    AutomaticCuration,
    AutomaticCurationParameters,
    AutomaticCurationSelection,
    CuratedSpikeSorting,
    CuratedSpikeSortingSelection,
    Curation,
    MetricParameters,
    MetricSelection,
    QualityMetrics,
    UnitInclusionParameters,
    WaveformParameters,
    Waveforms,
    WaveformSelection,
)

# We also import a bunch of tables so that we can call them easily from spyglass.common import ( RawPosition, HeadDir, Speed, LinPos, StateScriptFile, VideoFile, DataAcquisitionDevice, CameraDevice, Probe, DIOEvents, ElectrodeGroup, Electrode, Raw, SampleCount, LFPSelection, LFP, LFPBandSelection, LFPBand, FirFilterParameters, IntervalList, Lab, LabMember, LabTeam, Institution, BrainRegion, SensorData, Session, ExperimenterList, Subject, Task, TaskEpoch, Nwbfile, AnalysisNwbfile, NwbfileKachery, AnalysisNwbfileKachery, ) from spyglass.spikesorting import ( ArtifactDetection, ArtifactDetectionParameters, ArtifactDetectionSelection, ArtifactRemovedIntervalList, SortGroup, SortInterval, SpikeSortingPreprocessingParameters, SpikeSortingRecording, SpikeSortingRecordingSelection, SpikeSorterParameters, SpikeSorting, SpikeSortingSelection, SortInterval, SpikeSorting, AutomaticCuration, AutomaticCurationParameters, AutomaticCurationSelection, CuratedSpikeSorting, CuratedSpikeSortingSelection, Curation, MetricParameters, MetricSelection, QualityMetrics, UnitInclusionParameters, WaveformParameters, Waveforms, WaveformSelection, )

In [7]:

# Define the name of the file that you copied and renamed from previous tutorials
nwb_file_name = "beans20190718.nwb"
filename, file_extension = os.path.splitext(nwb_file_name)
nwb_file_name2 = filename + "_" + file_extension

# Define the name of the file that you copied and renamed from previous tutorials nwb_file_name = "beans20190718.nwb" filename, file_extension = os.path.splitext(nwb_file_name) nwb_file_name2 = filename + "_" + file_extension

First, make sure that the results of your sorting from tutorial 1 are stored in SpikeSorting table.

In [13]:

SpikeSorting & {"nwb_file_name": nwb_file_name2}

SpikeSorting & {"nwb_file_name": nwb_file_name2}

Out[13]:

nwb_file_name name of the NWB file	sort_group_id identifier for a group of electrodes	sort_interval_name name for this interval	preproc_params_name	team_name	sorter	sorter_params_name	artifact_removed_interval_list_name	sorting_path	time_of_sort in Unix time, to the nearest second
CH6120211203_.nwb	1	raw data valid times	franklab_tetrode_hippocampus	Michael Coulter	clusterless_thresholder	default_clusterless	CH6120211203_.nwb_raw data valid times_1_franklab_tetrode_hippocampus_ampl_1500_prop_001_5ms_artifact_removed_valid_times	/stelmo/nwb/sorting/CH6120211203_.nwb_raw data valid times_1_franklab_tetrode_hippocampus_d02e4cc4_spikesorting	1652421556
CH6120211203_.nwb	1	raw data valid times	franklab_tetrode_hippocampus	Michael Coulter	mountainsort4	franklab_tetrode_hippocampus_30KHz	CH6120211203_.nwb_raw data valid times_1_franklab_tetrode_hippocampus_ampl_1500_prop_001_5ms_artifact_removed_valid_times	/stelmo/nwb/sorting/CH6120211203_.nwb_raw data valid times_1_franklab_tetrode_hippocampus_3ce905c4_spikesorting	1652390802
CH6120211203_.nwb	2	raw data valid times	franklab_tetrode_hippocampus	Michael Coulter	clusterless_thresholder	default_clusterless	CH6120211203_.nwb_raw data valid times_2_franklab_tetrode_hippocampus_ampl_1500_prop_001_5ms_artifact_removed_valid_times	/stelmo/nwb/sorting/CH6120211203_.nwb_raw data valid times_2_franklab_tetrode_hippocampus_3236edfe_spikesorting	1652421602
CH6120211203_.nwb	2	raw data valid times	franklab_tetrode_hippocampus	Michael Coulter	mountainsort4	franklab_tetrode_hippocampus_30KHz	CH6120211203_.nwb_raw data valid times_2_franklab_tetrode_hippocampus_ampl_1500_prop_001_5ms_artifact_removed_valid_times	/stelmo/nwb/sorting/CH6120211203_.nwb_raw data valid times_2_franklab_tetrode_hippocampus_175d4957_spikesorting	1652391136
CH6120211203_.nwb	3	raw data valid times	franklab_tetrode_hippocampus	Michael Coulter	clusterless_thresholder	default_clusterless	CH6120211203_.nwb_raw data valid times_3_franklab_tetrode_hippocampus_ampl_1500_prop_001_5ms_artifact_removed_valid_times	/stelmo/nwb/sorting/CH6120211203_.nwb_raw data valid times_3_franklab_tetrode_hippocampus_0aa65847_spikesorting	1652421651
CH6120211203_.nwb	3	raw data valid times	franklab_tetrode_hippocampus	Michael Coulter	mountainsort4	franklab_tetrode_hippocampus_30KHz	CH6120211203_.nwb_raw data valid times_3_franklab_tetrode_hippocampus_ampl_1500_prop_001_5ms_artifact_removed_valid_times	/stelmo/nwb/sorting/CH6120211203_.nwb_raw data valid times_3_franklab_tetrode_hippocampus_b3b560dc_spikesorting	1652391465
CH6120211203_.nwb	4	raw data valid times	franklab_tetrode_hippocampus	Michael Coulter	clusterless_thresholder	default_clusterless	CH6120211203_.nwb_raw data valid times_4_franklab_tetrode_hippocampus_ampl_1500_prop_001_5ms_artifact_removed_valid_times	/stelmo/nwb/sorting/CH6120211203_.nwb_raw data valid times_4_franklab_tetrode_hippocampus_a248a29c_spikesorting	1652421729
CH6120211203_.nwb	4	raw data valid times	franklab_tetrode_hippocampus	Michael Coulter	mountainsort4	franklab_tetrode_hippocampus_30KHz	CH6120211203_.nwb_raw data valid times_4_franklab_tetrode_hippocampus_ampl_1500_prop_001_5ms_artifact_removed_valid_times	/stelmo/nwb/sorting/CH6120211203_.nwb_raw data valid times_4_franklab_tetrode_hippocampus_52ff93db_spikesorting	1652391770
CH6120211203_.nwb	5	raw data valid times	franklab_tetrode_hippocampus	Michael Coulter	clusterless_thresholder	default_clusterless	CH6120211203_.nwb_raw data valid times_5_franklab_tetrode_hippocampus_ampl_1500_prop_001_5ms_artifact_removed_valid_times	/stelmo/nwb/sorting/CH6120211203_.nwb_raw data valid times_5_franklab_tetrode_hippocampus_c75bf7f7_spikesorting	1652421799
CH6120211203_.nwb	5	raw data valid times	franklab_tetrode_hippocampus	Michael Coulter	mountainsort4	franklab_tetrode_hippocampus_30KHz	CH6120211203_.nwb_raw data valid times_5_franklab_tetrode_hippocampus_ampl_1500_prop_001_5ms_artifact_removed_valid_times	/stelmo/nwb/sorting/CH6120211203_.nwb_raw data valid times_5_franklab_tetrode_hippocampus_962a4aa4_spikesorting	1652392074
CH6120211203_.nwb	6	raw data valid times	franklab_tetrode_hippocampus	Michael Coulter	clusterless_thresholder	default_clusterless	CH6120211203_.nwb_raw data valid times_6_franklab_tetrode_hippocampus_ampl_1500_prop_001_5ms_artifact_removed_valid_times	/stelmo/nwb/sorting/CH6120211203_.nwb_raw data valid times_6_franklab_tetrode_hippocampus_11b507a6_spikesorting	1652421841
CH6120211203_.nwb	6	raw data valid times	franklab_tetrode_hippocampus	Michael Coulter	mountainsort4	franklab_tetrode_hippocampus_30KHz	CH6120211203_.nwb_raw data valid times_6_franklab_tetrode_hippocampus_ampl_1500_prop_001_5ms_artifact_removed_valid_times	/stelmo/nwb/sorting/CH6120211203_.nwb_raw data valid times_6_franklab_tetrode_hippocampus_ca67c53d_spikesorting	1652392496

...

Total: 4702

sortingview web app¶

Currently (June 2021) the manual curation of spike sorted data is done on the sortingview web app. To proceed, click on the link printed at the end of spike sorting in tutorial 1. Alternatively, run the following cell and click on the URL output.

In [17]:

# workspace_uri = (SpikeSorting & {'nwb_file_name': nwb_file_name2}).fetch1('curation_feed_uri')
# print(f'https://sortingview.vercel.app/workspace?workspace={workspace_uri}&channel=franklab')

# workspace_uri = (SpikeSorting & {'nwb_file_name': nwb_file_name2}).fetch1('curation_feed_uri') # print(f'https://sortingview.vercel.app/workspace?workspace={workspace_uri}&channel=franklab')

This will take you to a workspace on the sortingview app. The workspace, which you can think of as a list of recording and associated sorting objects, was created at the end of spike sorting. On the workspace view, you will see a set of recordings that have been added to the workspace.

Clicking on a recording then takes you to a page that gives you information about the recording as well as the associated sorting objects.

Click on a sorting to see the curation view. Try exploring the many visualization widgets.

The most important is the Units Table and the Curation menu, which allows you to give labels to the units. The curation labels will persist even if you suddenly lose connection to the app; this is because the curaiton actions are appended to the workspace as soon as they are created. Note that if you are not logged in with your Google account, Curation menu may not be visible. Log in and refresh the page to access this feature.

In [ ]: